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Three questions were addressed in an experiipent in which subjects followed 
instructions xo complete tasks Involving several pieces, of electronic 
equipment: (1; Two instruction fonnats were^ compar<sd; a'T^ierarchical menu 
format containing natural chunks of instructions was not superior overall 
to a simple st&p-by-step instruction format* Ihe rri^inu fo.rmat was superior 
■ only if the subject was familiar witK the type of device, and waB sometiyies' 
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subsCant^talXy inferior otherwise* (2) Experts were compared to nonexperts^ 
-and found to -be fastef'overall^ and able ;fo operate e'qulpment with fewer 
instructions in^Kfe iffenu ccAiditi9n. They^were also faster when complex 
physical actions Vere involved. Thiag^ there- were both specif ic and general 
effects of expertise, O) Evidence wks sought that knowledge^o^ how to 
'operate equipment was echemtic. It was expected" that when i^ubjects^in the 
menu format condition^perated a" device without selecting any instructions 
to read, ^heir sequenc^e of actions should corresppnd ^t|> stereotyped schema^' 
-llke'patterns. .This occurred only veakXy^ suggesting that even exp^'ts 
J pperate everyday devices ift a problem-solving luode^ rather than by retrieved , 
complete procedures, t ■ . , 
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Abstract. 

^Three questions were addressed, -in kn experiment in^^vhich' 
subjects followed instructipaS to complete tasks involving peveral 
pieces of electronic equipment; (1) Two instruction formats \were' 
compared; a hierarchical menu format containing natur^ chunks' of ^ 
instructions was not superipr overall to a' simpVe ^step^-by-step 
instruction format- The, menu format was superioj^only^'i^f the 
subject was familiar with the of ^device, and was sojaetimes' 

substantially inferior otherwise* (2) Experts were , compared to 
nonexperts, and found to be faster overall, and able to operate, 
equipment with fewer instructions in the. menu conditdon^; Th^ey 
were also faster wh^ complex physical aotioi^s were 'involved 



)hus 



there 



were bo*h specific and general effects of 'expertise * 
^as sought that knowledge of Kow to operate equipment 
It was expected th^t when subjects in th^ menu 
■ operated a device wi^hout^ seletst ing ;any 
i^i^tructions to reeid , "their sequence of actions shoold-^orrespond 
to stereotyped schema-like patterns., ^t-his occurred only weakly, 
suggesting' that e^en expert^ ijperate every^Jay, ^devic^s in a 



(5j Ev'idence 
was . schematic 
format condition 



problem-solving 
procedures * 

■ \ 



mode 



rathe^r . than by |*et rieved complete 
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How Experts and* Nonexperts Operate " 
Electronic Equipment frdm Instruct^ ions 



David E. Kieras MarkT!ibbits , 
and* 
^usan Bovair 



'This ^eport^descrlbes results from an experiment which was 
designed to assess three questions, about *how people operate a 
'place of equipment from written instructions. The quest icms ^ deal' 
with instruction format, expertise, and the organisation^of prior 
knowledge, in' a task in ^ which subjects ^ must - follow a set of 
instructions in order to complete a task involving an electronic 
device, ' ^» , 

^ ^ . i ' ' 

,The first question is one of instruction format (see Smith & 
GoodDan, 1982)., This is the difference between whether the format 
or laj^out of the -instructional material forces the user tp execute 
each step in order, or whether the instructions^ allow .the user' to 
pick a^nd choose the material to b6 read ar)d executed- In - ^this 
experiment, one group received step-by-step instructions that were 
jjresentedla single step at a t ime , and 



the 
received 



subj ect had to " read 
a . hi enarchical menu of 



ERLC 



every step - The. other group _ _ 

instructions, in which the subject could^^ither execute the task 
with only a high level* description, or could request more detail. 
In this way, t(?e sub ject" would only^ have ^tp .read th^ :- insti*uctions 
that he or she felt was necessary, to epcecute the task- The 
rationale oX this manipulation is, that an expert Subjeet could 
take advantage of the " hierarcfS^l menu format , because large 
portions pf the task would be familiar. ^ ^fiowever, a, nonexpert 
subject would have to,^ead all of the instrdctiori^^ anyway, so the 
menu would not b§ of atny great advantage- ^Purt'hernjore , there 
should be' relatively , little -difference between experts and 
iQnexperts on step-by-sterp,. inst'ructioris , because in botli cases all 
/of the steps must be read. . . ■ ► . . . - t 

The second question is - the nature ■ of expertise effects - 
While- expert ise ^ has been ' heavily studied (^see Chi, Peii^ovlch , & 
7:.^ser, 1981 Chi (S£>^Glaser, in prejss), it has not beeV. examined 
in *ne q^ntext of operating electronic equipment, a domain of 
greaj* pricticaL* importance-. Generally, it is expected that 
experts w.ould complete the tasks Taster, and read fewer svips in 
trie menu condition, Howeyefr, thi-$ could' depend on the device 
under 'consideration- Only experts woald ^be familiar with some 
devices*, but even tne nohexpertg should be able^to operate^ other 

Likewise, 9ven nonexperts should 'know sorae 'things 



devices easily 

about almost any device , such as h6w' to turn it on. 



'hus , it was 



expected that there .wQuld .be ^n interaction of subject expertise, 
experience with the,'e'xa^t deviW, and the nature of p-^rticular 
steps in the inatruc-ti ons ^ [ The basic ques-tij^n about expertise 
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effects is whe^tKer th^re are general effects of expertise, or 
whether they 9,re specif-ic to the in^diviUual devices involved, ^or 
this reason, several devices" of , widely diffsring- familiarity were 
used. i ^ 

The third question concerns the nature of the prior knowledge 
that subjects have about devices . - In Kieras { 1 9S2) it was 
^ggesited that knowledge of devices Is organised in the ' fbrm^ of 
schBmas. Thes^ scHgmas would, include knowledge not only of how to 
recognise a particular type of devii;e> but also, its - typicg^l 
3t ructure artd operating procedures . If device knowledge is 
organised by sohemas , there should be clear patterns in the data 
obtaiaed in this experiment* Menu choices should follow patterns 
tbSLt would be expected from schematic deyice knowledge . If 
subjects operated the equipment entifely^ from prior knowledge > 
without r«iad ing instructions , which happened' in many^ cases,,- then 
their behavior" should follow some pattern that can be described In 
terms of device schemas* * ' 



iriii 



The tasic manipulations performed in this study were as 
fpllows: >several devices were used, wMch included two every-day 
devicfes, twn devices familiar to only experts, and two n6vel 
devices^ familiar to neither experts 'nor nonexperts. The subjects 
were either experts, who typically had several years of working 
experienc'e in Electronics , or nonexperts , who were ordinary 
college students- A ^ questionnaire was used ' to confirm the 
sulij^ct's classification, and to assess each sub;3ect's experience 
wi th the individual devices used in the experiment , The two 
instruction formats were either a step- by-step 'format 'or a 
hierarchical' menu format. The te^rminal - nodes > of the menu 
hierarchy consisted of the exact same individual irtstruction steps 
as were used in the step-by-step format. The variables measured 
were the total completion time for each task on each device, the 
completion time for each individual step in the step- by-step 
instructions > and in zhe menu condition, ' the^ individual menu 
choicer, and tneir completion times*' The subjects^' behavior was 
I recorded ^on videotape to allow detailed scoring on the sub^Jects* 
atctivities while performing the tasks. 



METHOD 



.Mater ials 



f , 



Devices. The six devices used are described in Table- 1 



The 



radio, cassette recorder, VOM, and oscilloscope were of a standard 
aalce* The phi, phenomenon demonstrator was professionally built > 
tut in general construction^ style it appeared to be a "home-br6v*' 
^Ateiir, job. The physiological stimulator is a stdbftdard piece of 
apparatus" in a physiological psychology lab, but as the ratings 
confirmed, it was essentially unfamiliar to all subjects. l^otice 
thaV all . of ^ tne Don-everyday devices were relatively 
old-fashioned t being from the vacuum-tu&e^ era'. The, <levicea were 
prepared before presentation to 'each sub J ect by setting all 
controls to incorrect positions 'so that in order to, complete 'the 



Tabie ^1 

Devices Used in the Experiment 



Device 
1 - Radio 

2. Recorder *' 
5- VOM 

4 - ' Oscilloscope 



5 * PKi Phenoflienon 
^ Demonstrator 



6* Phjcsiological 
Stimulator 



Description 



/ por'table AM-PM'radio*^^ with ^built-in AC 
' adapter , antenna, volume , tone ^ tuning, 
and band controls* , 

• A portable aiidio cassette tape recorder, 
with keyboard tape controls, red record 
Interlock key^ and volume control • Supplied 
cassette was not fully rewound 

A standard 'volt-ohm-milliameter, with a 
supplied resistor to measure. 

A dual-trace triggered-sweep oacillosxxope 
with standard audio signal generator and 
connecting cables; 

A deVice that flashes two connected neon 
bulbs alternately at various rates and phase 
relationships * ■ . 

* 

A large device with several diel-multt^lier 
s^i^ "that produce jpuls^s of specified 
magnitude^rate and duty cycles; a neon . 
bulb ie connected fo the output to indicate 
the pulses. 
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■task, each control would have to be properly set. • 

i; ' ' ' - ^ 

Instructions . A major goal in composing the instructions^ was 
to allow the menu and; the step-by-St^p i^istructions to' be easily^ 
comp'axed to each other." This was don^^y preparing the materrals 
so"^ ^that the terminal steps in t]ie menu instructions were "exactly 
identi^l to the steps comprj^sing the step-by-step ^ instoructione., 
and were Horded and displayed identically. * ' * ^ 



Thjp menu Instructions made up a hierarchy of natural "chunks" 
of .the operating procedure. .Determination of the chunks was .done 
intuitively. It Is clfear* f rom some aspects of the results tjiat 
some of the chunks chosen" were in fact natural unirts; however,^ 
the data do not definitively q^onfirm the ^chunk classification. 

Each set of , instructions beg&n with a statemejit of the task 
that ,the subject had to accomplish. This main task statement was 
specific enough that the subject could, if he or she had adequate 
pr ior knl3wledge > complete the enti re ^task from just th is 
statement. However, the main task statement did not descMbe how 
the controls on the device, had to "be set or operated. Table 2 
lists thfe'tasks Jhat were to be performed on each device, in the" 



same wording as 
Subjects 



;hey were shpwn to subjects. 



The nonexperts were - recruited by 'campus and newspaper 
advertisements > and were paid $5 for |i^rt,icipating. As shown 
the experience questionnaires adminietere3^o the subjects, only 
one expert subject was inadvertently recruited by this method. 
The expert subjects were^ecrui ted -by advertisements directed at 
electronics experts. ^n all cases, the subject's obtainjsd were 
highly .experienced in electronics; the typical expert had- several 
years experience as an je-lecjtronics technician in the military. 
Twenty subjects were recruited by each method, but in the analyses 
used below, the -classification w^s corrected', to ^yield nineteen 
nonexperts and* twenty-one experts. Since earlier studies seemed 
o ' suggest that there were strong sex differences, among 
nexperts, and female electronics experts were extremely hard to 
locf^te, all subjects jised. in this .experiment vere pale. 

Design * The instruction format condition was . determined at 
random for -^f'ach subjegt. Each subject carried out -the six tasks 
on tne six devices in the. same instruction format condition, ^ ^he 
device tasK« vere done in a fixed order", which is th^ ordfr in 
which" the dfevices a/e listedX in Table 1. This order was chosen to 
presejit the tasrks and devaces in order ^of decreasing fam'iliarity, 
and increasing apparent complex ity "within each . level of 
,j familiarity . 
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Table 2 



.Task 
1 . ■. 

2. 

3. 

4. 



Main Task Statement 



5. 



* J^^**^*^-*— — — — — --^ — — — — — — *— — 

. listen to 'Station KUAT-FM (-90.5 FM) at medium volume^on , 
the portable fadio; ^ . - ■ 

Record the words "testings .1,2," 5" on the 'oasse^tte , 
recbfder ,1 and play the words .baclc at medium volunie. ■ 

Measure the resistance', of the resistor vising the volt-ohni 
meter. 

Use the signal generator ahd"^he oscilloscope to displajt, 
about two AC wave cycles on theioscilloscope screen. . 

l/se the phi phenomenon demphstrator to f^ash the lights ^> 
^at ^ CPS (cycles per* second)- /' 

Use the stimulator to flash the neon light at a frequency 
* of l^CPS'(cycles per second) with a flash duration of 
*7 seconds and a delay of .5 seconds. 
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\ ^'^ • 

Apparatus and Procedure - * ' 

Bach subject vas run individually, and vas seated in a ^mail 
rooia before --a table. On ^the right-hand end pf the, table was a 
standard video terminal, on whioti a laboi'atory cbmputer^ displayed 
the InBtructians. The . left-hand p6rtion of the tabl« was occupie^d 
by the ^device. A videotape recorder r^ecorded all of the *sul)ject'^ 
a5tivi4;y"- The instructions were presented* ohe_. step or menu ett a 
/time, with the subject tapping the space bar "or .tjjpin'g a' choice 
/ number 1:o go on to tlve next , d is^lajr . The laboratory computer 
^ recorded the amount of tim^ that the subject left each instruction 
step or menu, on the screefi. Due to the- nature of the equlpmerit, 
Y aRd the prohibitive scoring effort involvj^d, it 'Vas not practical- 
^ to distinguish the time the subject T^pent reading from the tlm;e^ 
the subject spent carrying out the instructions. 2'^^®^ ^ 
labo^'atory computelf^ was ab.le x>!ily 'to record the completion time 
' for, each step, defined as the total reading jplus execution time 
for the instruction step- The^videotajpe recording was used to 
determine what subiects actually did on each step. ■ " 
/ r- - . . 

The ^devices were l>rought into th.e.room one at'a'time, and the 
subject then carried out the task "on the dev.ice. Whfen the*subjec't 
had reached the end oi 'the Inst rue tl^ffrwr-'fTie exp&rimenter Returned, 
and checked that the task had been carried out* correctly , in terms 
of vjKhet'her .the final correct result was achieved. The device was 
' then ,removed,' and a new device brought in. Subjects *who did not 
. achieve the proper ^inal result were asked to repeat the task; 
J however, the dita fl^om^ th^se repeated, tasks were later dropped 
^from the analysis. ^ * - ^ , ^ 

Due to inadequate training of .the experimenters, on some 
trials the "-equipment wa^ being moved in* arid put of the room while 

' the cloclc was running, .making the' completion time feodvA of the 
first ^«i*rstruction unreliable. ■ It is belie^^F^ that"' these Events 
are'not confounded with any of the experimental. -manipulations, so 
the analysis of ^ total completion time'wpuld be conservative due to - 

" the ,extrg-neous variability. Examination of the video tapes shows 
/?iat the subj ects -were ' visually inspecting, the devic?^ ■ while .they 
Iwere be ing brought in, aod so tjj,ese times reflect the tc?tal time 
^hat the subjects ^in^eracted' with the devices to cot^pl^te/f^e 
task. ^ ' ^ . ^ " ' 



To tag Compl|j:ion Tim^^- 



Analysis m^hod;^ . The total* completion time-' for eaOh -subject 
on ea9h ^ task.>was calculated as the total ^elapsed - time from the 
preaentatioo of the main ta^k statement, until the experin^enter- Ijad 



om tasks 
re ^ than 



confirmed that the task was completed tJorrectl^. ' D^ta 
wer*^ propped Iti wftich the subject did the ynt'ire task m^ 
onrfl* or failed to do the task at alj, correctly. , 0,ut of the total 
of.^^40 task attempts, 14 were thus dr9ppedL. Dufe to th^ unequal 
group sisfes., missing data, and^ . unbalanced device experience 



f ac"fer t the total times were ^alyaed using stepwise multiple 
"regressi-on.' ^ *■ " , * , , " . 

■JJhe subject's expertise group, - instruction format condition, 
andx^ subj ect* s experience with the 'iniiividual device wJfere 
" represented as dummy variables . The device experience variable 
was based on the questionnaires' that each subject^ filled out. If 
the subject indicated ^ny actual usage experience with the device, 
then. the devi-^e experience dummy variable received a value or on.e; 
otherwise a value of aero was ^assigned . The device factor- -was 
flpresented as a set of five'dummy coded variables with the radio 
being used' as the baseline? Pillowing, the methdd suggested by 
Pedhasur (1982) for mixed designs/ a variable whose value' is the 
subject^s mean total completion time over the" six devices was 
included . The betweeii-subjects factors and interactions' were 
entered first in the equation, followed by the subject's mean time 
variable, followed ,by all of, the ^ wiXhin-subject factors an^ 
interactions^ The analysis, was ^hie]^c]iic3,l, in that main effects 
were forced, into xhe equation before interactions. 

All of the interactions between, .subject experie^nce, device 
experience , and , i^nstruction , format condition werfe 'represented , bi^.t'^ 
only inslTHiation^f ormat condition and subject expertise group wer^ 
allowed to interact with il\e de^ce factor; device experience was. 
not allowed ;to interact witn the device factor'.^ The rationale for 
this decision is that the device experience Variable is alTready 
specific t(5-^ndi vidua! devices, so" interactions between individual 
device e?ygerience and individual device dummy variables would be 
difficult to interpret. v ^ . 

Note that su^^ject expertise and specific device experience in 
these data are only . sli^htT^"-^or related (jr=: • 1 3) » and * the 
interaction. be,tveen subject expertise and device experience ws^s 
not significant. Thus' these two factors m^e practically 
ind^pendent^wContrlbi^tiQns to the total conlpletiQji times. Two of_ 
the 'devices were familiar to jfev^ryone, and . two were ''unfamiliar to" 
-almost everyone, resulting in these- two variables being* nearly 
orthogonal! ' > ^ 

. Xith a total of .25 variables in the equation and 163 degrees 
of freedom in the residual, 81.5^ of the variance .in the total^ 
completion times was accounted for. This extremely high figure is/ 
due to two factors: the subject'^ mean completion time^ accounted 
for appr<3^x'imately (t£ the variance, and the ^eviie factor 
accounted^Nfpr about 50j6 of the variance. .This is clearly due to 
the^fact/^^Kat the devices .varied suljstaittially iti number of steps 
in thef tasks, and thus' the completion times' vary syst'ematic^al^-y 
over an extremely wide range-. The ^eff ects to be discussed below 
wereH all tested for significance "at the .05 level, using the 
"P-to-remove" "statistic, which is a. conservative estimate of the ■ 
significance of an individual v^ria"ble as if /it were the last to 
enter the equation.' ^ - ^' 
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, M9,in effects^ . Table 3 shows the itfeEi,ns for the various main 
effects^ that we^re significant. Tlife sub;3ect ^expertise variable was 
quite significant; efpcperts ii/ere • about one third .faster in 
completion time than; nonexperts/ , There was no significant main 
effect of instruction format condition > feven. though the menu 
condition ave^^aged about 30 seconds faster. This means that > 
counter -to Intuitiion, the menu format wa^ not reliably superior 
overall to the' step-by-step format. This is probably, a result of 
the fact that while fewer steps were read in the menu condition, 
more materiibl has to, be read irl addition to the individual steps. 
The device, experience fac^^or was significant;, being familiar with 
'^a specific device led to a 30^ improvement in" completion^ time . As 
would be expecteflj there is "a very g1;rong main effectVof device*s. 

Interactions . The interaction between .device exparience and 
instruction format condition, shown in- Table 4, was significant. 
The menu instrugtions are actua^I^ly slower than the step-by-st^ep 
instructions if the device 'is ncH; familiar, b^t substantially 
faster than the , Btep-by-step instructions if the device is 
familiar. , ThUs , not only do the menu instructions allow the user 
to take advantage of prior knowledge more than the step-by*-step 
inslfTUctions, but the lack of prior knowledge means that ,the extra 
"overhead" in menu -instructions, plus mistakes made as a result* of 
skipping instructions; actually slows down task. compl'etion\ 

The intera'fction of instruction format condition^ and device, 
whose meana are shown in Table 5» was significant. For the' radio^ 
recorder, and phi demonstrator > the menu condition produced faster 
resul'ts than the step-by-step condition. However , the VOM> 
oscilloscope, and stimulator produced the opposite effect. This 
is probably due to. the fact that the^e are devices which were 
especially difficult for nonexperts, exaggerating the effect " of 
'tKe extra material in the menu format ^ Table 6* shows the 
interaction between devices *and ^su'bj ect expertise group, which was 
also significant.^ He^ it is clear tha^t the oscilloscope and VOM 
were especially hard for the 'nonexperts compared to th^ experts. 

^ ' The three-way ' interaction between subj ect exp^r'fise, 

'corfdition, and device was significant, and 'illustrates the key 
result. The means ar.e shown in Table 7j,.-^hich includes the 
percent gain resulting, from using the menu instructions instead of 
s>tep-by-step , f5r nonexperts and experts .on each device. One 
clear result is that the experts benefit from the use of the mepu 
format on all devices except for the stimulator, where there is a 
subst^-ntial Impairment in performance. This is probably ^iCie to 
the fact that since this was a complex and novel device, the 
experts* a^ttempts to -operate it without reading much of the 
instructions often led them- down "garden p,aths.*' For example, one 

'expert plugged iridicator light into the wvong ;3ack, and then 
spendin^g a long time trying to set the controls to light it. With 

^the nonexperts, the two expert-familiar devices, the VOM, and the 
osx;i lloscope , produced much longer completion times in the penii 
instructions | compared to' the step-by-step- SJnce many of the 
npnexpert subjects claimed experience^ in usi^ng the VOM, t^eir 
longer , complelffdn times^ in the menu condition may be similar to^ 



>. ' ' Table 5 • . ■ 

Main Effebtg in Total Time Data, ' ' f 

Effect Means , Sighif^i^ance 

\ Expert iae ' ^ ^" ^ / - ft 

* Nonexperts ^ 514*0 ' -t ^ ^ ' ^ 

Ex'perts "'214.4*- * , " , . ^ - - 

. - \^->- 

^ DevTcS TBxD^riefrc'6 ^ _ ^ — - - 

Non-familiar 51 9 •4" ** ' /_ ^ * 

Familiar ■ .225.0 * ' , 

■ Inatruction f^ormat ' < ^sf . ' ' " 

Step-by-3tep ^ ' 274-4 ^ ^ ^ NS - ' 

Menu 247.1 ' . ' 

•Devices * : ^ * . * 

Radio ' ^7.^1 ^ , , — ■ 

Recorder , 166.2 * "I, 

VOH ' . 256.8 ^ , ■ ' 

^Oscilloscope ' " ' 511'9 - , . < t * ^ 

Phi Demcinstrator 118.8 , , ^ - - ^ ^ 

Stimulator - 545*? ■ ^ ' 

: ; t**-, , — ?— -1. — - 



I • V . . ' ■ Table 4 / 

Total Time (Sees) as a Function 
qX Bevice Expertise ^and Instruction Format 



'Device 'Experience 



Nof~ familiar > ^ 307.8- ■ 331 .0 

Familiar . ' , f 254-4 ' 185.1 



Table 5 

Total Time (Sees) as a Function 
of Instruction Pornrat and Device 









Device ' 






4 


PormUt 


Radio 


Recrdr 


VDM 


Oscil 


PhiDem 


Stim 


Step^by-step 


185.7 


, 201 .3 


230.1 


504.3 


210. .5 


314.4 


Menu 


88.4 


131 .0 


297.8 


521 .9 


167.0 


375.5 



m * 

Table 6 

Total Time (sees) as a Function, 
of Subject Expertise and Device 



DeyicQ , ■ ■ 

EJ^pertise Radift^ Recrdr VOM Oscil PhiDem Stim- 



Sanexpert 165-8 188.7 342.8 649-0^ 230.3 ^ 567.8 

Expert 111.0 ^ 145-7 175.8 396.4^151.2 3^1.3 



Instruction Format 
3t^i-by-step , Menu 



16. 



Table 7 ^ ■ 
Mean Total -Time for ekch Device,^ 
Instruction Format, an<i Expertise Cjroup; 



D.evloe 

Recrar 



Oscfl 



PhiDepi Stiffly 
— _ 1 _ 



Nonexperl7 ■ ' ' ■ t 
Step-ty-step 224.5. 216-5 - 270.8 604-5 
Menu' - 100.9- 158.2 462.8 725.6 



^ &ain 55^- 27^ -71^ -20^ 



260.4 t 559.5 
196.9 578.0 



24^ 



"5^ 



Experts' v 

Stfeir-by-steji- 147.0 186.4 189.5 
Menu ' 78.5 108.8 V3^/5 



i> &ain * 47^ 41?^ 17^ 



404.4 
587.4 

■ ■ 4^. 



160.6 
142.6' 



269.2 
575.5 

-59^ 




the "garden path" effect obtained for th^ . expe^'ts with" the 
stimulaWr. Hamely, , a liVtle, f amiTiarit^j" with ^ a devijfie Is a 
dangerous thing;- -it os^ l^ad to ' Ibnger cdmElet'^ion times if 
instructions are ^©"t followed • The elevated" time „for these 
subjects in tha menu_ condition with the osdilioscope is harder ,^to 
explain i " ' ■ ^ ■ ■ 

Conclusion* These", results demonstrate that the , virtues of 
the two instruction formats-lireTieavily d'ependent upop the user*s 
general expertise and also the famrli-arity with the specific 
device* In general, the Interactions ^eeip #to be due most3.y to the 
specific familiarity with the device, as opposed to the subject's 
general expertise. That is, the fact that the interaction between 
device experience and .instruction format was significant, but" the 
interaction between subjeci^ expijfrtise a^nd instruction format^^was 
noV, suggests that the advantage of menu instructions 
of -specific familiarity wit>i the device, . and 
expertise,. Electronic experts may not "do better witjh the mqnu 
Instruction format unless they have specific ^amiliai^ity with the 
"device in question- Alternatively, "if the device is unfamiliar, 
experts cap benefit from menu instructions \ if* "^he device is 
simple, such as the phi demonstrator, but not if\it is . compl^x^^ 
such as the stimulator* 

Dn the other hand, the significant main ^effect of subject 
expertise i " even with specific device experience taken into 
account, is important. Experts were, generally faster ' than 
nonexperts at operating the', " equipment, regardless of its 
familiarity. 0*her aspects of tiK?^ results suggest th'at this is 
due not just to faster execution of a<ition^, and also to better 
organised and more efficient actions as' well* 



Is a matxter 
not general 



Menu Choices 



Number of f raipes read > Table d shows the mean number of 
frames ( dispJays of instruction steps or menus) read in the menu 
cond it ion f o,r each group and each device * For example., both 
experts and nonexperts read only one frame for the radio, ?fe.mely 
the frame that contains the maiij task statement, but nonexperts 
chose to read an average of 50.4 frames of information for the 
oac illoscope task , while experts read an average of only 1.2. 
iJnese data were subj ected -to a multiple regression analysis 
similar -tro — the above one , with the factors being subj ect 
expertise, device experience, ' and devices, and interactions .of 
subject expertise with. device experience and individual devices 
were allowed *^^,The results are summarised in Table 9* > 

There were strong maio effects of device, with the VOM, 
oscilloscope I and stimulator requiring many more frames than ^the 
radio, which was taken as the baseline. The key results were that 
neither subject expertise nor -.Mevice experience, nor their , 
interaction, were significant predictors of the number of frame^^ 
read, OT\ce the main effects of device and th^ lliteraction of 
subject expertise with device were taken into account * A3 $hown 
in Table 8 , the VOM, oscilloscope , ph i demonstrator , \ Vnd- 
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Table 8' ■ 

ran iJumber of Frames Read In the Kenu Condition 
N * for Bach BxpeTti^e . &roup i^-^^ 



&roup , ' 
Nonex^pprts 
Experts-^-^^ ^ 



Raaio Recrar \. . VOM 
1*0 1*0 52*5 



1 .0 



t.o 



3.1 



Oscil 
50.4 
1 .2 



PhlDem 

' re . 2 

3.4 



Stm • 
47.8 

2.1 , 9 ' 



Table 9' «, 
iiegressioyi Analysis on Number of 
Frames Read in the Menu Condition 



Variable ' Coefficient 



CONSTANT * 10.1 

'SUBJECT EXP. - 0.0 

DEVICE EXP. -9.1 

DEVICE 2 0. 

DEVICE 5 . "■ 29.5 

DEVICE 4 4^ 46.5 

DEVICE 5 ""'TS* 9.1 

DEV^E 6 38.7 

SUB EXP X DEV2 0-0 

SUB EXP X DEV3 -26.3 

SU^ EXP X- DEV4 -45.3 

SUB EXP X DEV5 / -14. a 

SiJB EXP X .DEV6 M -25-. 9 



Sta. Coef 



.0 

-.25*1 

-.0 ■ 

.516 

.839 
.184 
■ .751 
- .0 . ~ 
-.360 
-■.618 
-.224 
-.-373 



P-to-Reaiove 



0.00 

?.99 

0. 
45.40 
113.19 
" 2.49 
43.98 

0.00 
20.82 
61 .52 

8.68 
25.15 



Notes 

R2 

used as 



is .S5 with 12 variables and N=107.' Device 1 (Baaio) 
the baseline for dummy coding of Device factor i 
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stimulator all required 



nonexperts . The main 
significant. Thus > it 



many fewer frames for experts than for 
effect of device experience was marginally 
is clear from the^e results that* the menu 
condition allows expelrts to benefit by permitting them to' read 
only a few frames. ; 

Choice Patterns. The specific pattern of frame phoices for 
each device was considered in terms of the menu hierarchy for each 
device. She intended organisat:j.bn of the menu instructiofis was 
that the levels in the hierajrchy would co^r'e'spond^ to the natural 
chunks in the operatioil of the device. However, contrary to the 
goals of -"the experiment > the evidence to support this claim is 
very limited in these data. Iri^order for 'th^re to be natural 
chunks in the oper'atlon of the device, the device must Iti familiar 
to the subject. However, if the device was fairly familiar to the 
sub^ec'*^ the sub;3ect would need to read very few frames, often 
only the main task statement fraiq^, and thus there would be few 
choices to reveal whiah " portions of the menu hierarchy were 
familiar and which ^were not,^ Perhaps different devioes would have 
yielded more useful data. ^ , 

However, there were some interesting patterns in the choices. 
Figure. 1 illustrates the bfest example. The figure shows the menu 
Kierarchy for the phi demonstrator in simplified form. The 
terminal portions of the tree consist of the sequence of actual 
steps that were identical to the step-byvstep instructions . In 
each box is shown the 'proportions of nonexperts and^. experts who 
tead the mater^aT in the box. Thu§, for example, the top-level 
box corresponds to the frame that states 4;he main^taslc- Almost 
all subjects then read the m^in menu which contains four items! 
powerinfg up the device, attaching the lights, .setting the mode, 
and ad;3u9tiiig the CPS dial- However, only 40^^ of the* nonexperts 
and 'only 10^ of the experts felt it was. necessary to get the more 
specific information about powering up the device,, and almost nctne 
of the subjects required the step-by-step instructions about how 
to plug in the device and turnClt on. The- other devices that also 
involved these steps also had^ this general pattern. Very ffew 
subjects, even nonexperts, required the specific 'i-nstructions: on 
plugging in and' turning on the device. This was true for the 
oscilloscope and signal generator combination, and also ^ true fbr 
the S'timulator , which was a very complicated and unfamiliar 
device. 

Another effect that appears in Figure 1 is ^the tendency for 
nonexperts to learn while doing similar acti^ties. notice how 
50?6 of the subjects required the step-by-step instructions for 
plugging in light A, but only lOjf of them went on to read the 
instructions for how to plug in light B. A similar effect appears 
in the - oscilTbsbope task, in which fever nonexpert subjects 
required the instructions ' for plugging in and turning [-OTr-'t"ne^ 
second piece of e^juipment than for the first piece of Equipment/. 
The obvious implica*tion of this effect is that subjects are n6% 
silnply executing ^he&e instructions as they read them, and thei) 
forgetting the instruction content when they proceed to tTtW next 
/ instruction. Rather, they seem to be able to take tjae content of, 
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Figure 1. Proportion of menu choices made at each menu level for experts (E) 
and nonexperts (N) on the phi phenomen<5n demonstrator. 
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one instituction an<i genBral'izre it immediately to apply to a 
similar situation. HoweveT^ these results are too limited to shed 
much light on' this issue. Further wo>k is" cleeurly needed. 

Step Completion Times ' 

** 

^Analysis method. * A rggression analysis was done ^tOt determine 
wh ich factors predi ct.j^the amount o:^ t ime taken tcf comple'te 
individual stej^s in the etep-by^step condition. ^ The - videotape, 
scoring was used to eliminate the times for individual steps that 
weV'fe defective. In addition, *the timep on the very first step, in 
the^ instructions were not included since *in" some case?, these 
tiiaes were oontaminated as described aboye* ^. This left a total of 
5008 \ indp/idual step times for the .analysis . Each instruction 
step was classified according to a set of categorizes, .shown in 
Table - 10, whiah are the general .types of actions stated by the, 
instructions. These categories we^;^ each represented by a dummy 
variable, vith thfe ISIMP category Wttig u^ed as the baseline. The 
video tapes for ea*bh subject were scored according to the action 
actually carried out by the subject on each step. The scoring 
categ6rie3 for th€ aQtions are shown in Table 11. These were also 
represented with^ dummy variables , with SKIP being used as a^ 
baseline . In order to ^ examine the chunking jtoroper ties of the 
step-by-step instructions, the variab^ MEKffwas defined, which 
reflects the proportion of t imee thei; subj ects irt - the menu 
condition read the corresponding step*-^This variable fbok on a 
value that depended on whether the sut)ject Vas an expert or a 
nonexpert. If the subject was an expert-,^'then the value of MENU 
was the' proportion of experts that viewed the corresponding step, 
in the Euenu condition. Likewise, for * nonexpert, the HEH^U 
variable was the' proportion of nonexperts that viewed that 
instruction. 



c 



ERIC 



An additional variable that reflected * properties ■ of the 
Instructions was the number of words in each instruction. This 
vaViable should ftot be taken^ to reflect comprehension time, since 
its coefficient is far too large;' rather, it provides a crude 
measure of the overall amount of info'rmation in the instruction. 
Additional variables entered into the analysis were l;he subject's 
expertise group, and tne jJSViCje experience variable, as described 
above- The subjec^t Vxpertise variable was allowed to interact 
with all of the instruction characteristic variables and the 
actiois, variables* As before,, the order of entry in the stepwise 

P-to-^remove** 
mixed design. 



analysis was hierarchical , and the conservative 
stati-gtic is reported. Pirially, sfaice this was a 
'the subject expertise* varialile jtas entered into 
first ,' followed by a subject mean variable, 
within-subjects variables* 



the equation 
then by ' the 



time results. A summary of the analysis is shown in 
Note that the coefficients must be interpreted in terms 
that all other factors are in the equation* There was 
a substantial effect of subject expertise (SUBEXP) , 
experts read the instructions on the order of »1 .6 
per .step than nonexperts . Also, the step 



Step 
Table'T?? 
of the fact 



in^hich 
seconds faster 
times differed 
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Dummy 



Variables Usea* to Code Instruction Conteats 



Variable Description"Una Example 



I LOG 
I AD J 
I3IMP 
I EFFECT 

ICOMPH 
^EXPH 



locate a pirt of tlie- device (locate the power switch) 
Setting a. control (turning knob to DC) 
A simple ^action (flipping a swittih) 
Adjusting a knob to produce a certain effect 

( aeroing ohiils scale) 
A complex physical action (plugging in a cord) 
A complect physical, action^familiar to an expert 

(seroing a mfeter) 



Table 11 

Dummy Variables Used to Code Subject's Actions 



Variables 

DO ' 
SKIP 
LOOK 
LOC 

ACT 



Deg<;ription 



Action sam'B as instruction f 
rfo action carried out 

Subject looks at device ^ ^ 

Subject "Ideates" a, part of device 

(e.g. touches it)" ^ 
Subject engages^ in some action other than £ibove * 
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Table 1:2 
RegressLarr Analysis on 
Completion Tim'es for e?Lch Step 
tlie Step"by-3tep Condition 



in 



Variable 


Coefficient 


Std. 


Coef, 


P-to-Remove 


CONSTANT 




-1^.45. 








SUBEXP 




i 


'-^110- 


36.96 


Smean 








■ 242.29- 


DO . 


■ 4.11 




V 9.54 


IOC ■ 


2.46 




.139 ' 


4-55' 


ACT 


7>30 




.188 ' 


129.20 


HOC 


1 .86 


1 


.126 


. ' 5.43 


'lADJ 


4.60 


i 


.116 


45.99 


lEPPECI?- 


3.f6 




.093 


18. 92 


WORDS ' 


.57 




'.339 


366.98 


ICOMPH 


.14.46 




.247 


154.99 


lEXPH 


6.42 




.156 


42.66 


EXPCOM . 


-5.52 . 




-.065 


11.19 


EXPXPH 


. -5.78 




-.100 


24. ?1 . 


HEKU - 


1.71* 




' .091 


8.39 



Notes 

R*2 is/-4075 with 19 variables and. H=3008. ^Pive variables 
are not shvwrj beca-use the P-ratios were nonsignificant. See text 
for explanation of variablef. Values for SUBEXP are based on only 
SUBEXP in the ^ equation, before SMEAN and the vithin-subjects 
variables are added. 
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substahtially depending , both / on-'*l>nich lactions , that subjegts, 
actually performed, and also in the properties of t^je instructions ' 
themselves. ■ This result "in itselff is. not too surprising. ^ 
However; it is noteworthy- that two of the stron^estr (as ahown bjLT 
the standardised regression coefficients) instruction factors ,ate ^ 
the number, ot words 'in the instruction (WORDS); and vhethe^'the' 
instruction ratitllred a complicated pHysical activity * (iCOjlPH) . , 
Instructions that required physical -activVties that are familiar 
only to experts, such" as adjusting th^ zero adjust sc<re.w on, l!he ^ 
V3M (lEXPH), also took significantly longer, -even'though such 
cases were fairly rare. ; ' 

4 

/She key resolts are the interactions. Qf_ expertise with two^ of, 
the " instruction characteristics , namely, complicated physical 
activities (EXPCOH), ' and" 'expert physical activities (EXPXPH). 
This suggests that not only are exper'ts faster across the board, 
but they are especially fast at certain complicated physical 
activities. "Informal observation "of th^ video tapes seems to 
confirm this. N6nexpert subjects often spend a lot of *^t,ime 
fumbling with cords ahd connectors^ while experts seem to know 
-fexactly what they are doing in these physica-1 activities, and 
proceed smoothly and^ precisely . 

An adcTitional kej result^ is that the MENU variable was 
significant. The .coefficient means that with all other" factors in 
the 'equation , a step thdt -was always read in the menu condition 
took about 1.7,* seconds longer than one that was never read." 
Assuming that the menu choices reflfect* the ' familiarity of 
procedure "chunks," the amount 6f time taken to complete^ a step is 
thus a function of its predictability on the basis 'of prior ^ 
knowledge . ' ^ ' 

Knowledge-ba^gd Operation ' , 

In Kieras (1962) ,it was proposed that people's knowledge of » 
electronic devices is organised- as a-hierarch^ of schemas, which" 
would contain, among other things, schematic information on ^ how to 
operate ^ the corresponding class of. 'devices. It is^ natural /tO' 
suppose 'that j.ust Sls ^a-v story schema /Specifies the order of 
appearance of items in a story; that a device schema would specify 
the order of the steps for operating the device. Thus, when 
subjects dperate a device ba6ed only on their knowledge, there 
would be a stereotyped seq^uence of behavior corresponding to the 
procedurffli schema for operating the device. Some of the data from 
tne menu condition was suitable for examining this issue; there 
were/ many cas^s where subjects attempted to operate the device 
after receiving only the main task statement; without req^uesting 
further insi;.ructions .- 

Analysis . method . ^ The videotape record of the subjects' 
behavior .was scored in terms of the individual activities that 
subjects performed i such as operating a certain control. "Data 
wer# dropped for subjects who got confused in the task or did it 
incorrectly in some way that w^uld^ invalidate the data. Both 
experts and nonexperts all operated the radio and cassette 
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recorder without any further instrMctionS in ^xhfe'^) menu* conditipn . 

Seven experts on 'the^ radio, and ninfe nonexperts - wWe thus 

available. ^^For th^cassette* reorder there j/ere". .eight .usable , . 

bebavior sequences^ from each* , grou^- With tliree other devi^es^ 

only experts operated "tUe device without instructions*, Por^ .the ^ 

VOMt , Q^scilloscope and signal generator combinati-en/ ^d the phi / - 

phenomenon demonstratoV , there wCTe five, eighty .ajid eight such 

sub j ects . , . .V ■ / * , * , ■ ' ; ■ 

The^ method of -analysing this sequence data was t^p ^ locate* ' 
sequences of sfetivities that ;t)ccurred^ at -le^^'^tfic'^ » and then"-^, ^ 
express tfae, s^ftwancea that Bjatr^St^ performed with-as few terms S9t 
possible .by * referring^ to. these conimor)^ sequences.. Mg^e 
specifically, the sequence data was represented as a transition 
n'Sr^work , tree diagram; in wh ich ' the nodes i;e?resen1; either , ; 
individual actions 'or action "subroutines/* and" gt^'^'single ,path* 
through the trfee diagram represents the activities of a single ^ 
subject. .See Figure 2 for an example. B§ich actioii is' represented 
^ by a two-letter symbol, and action "subroutines by -combinatiTtfris of , 
' .these symbols/ The depth of combination is 'indicate by "the 
notation; concatenated '^_syjnbols ar^p" the shallowest leve^^ with 
brackets and parentheses ilidicating aeepfer levels o^ 'Su'brouti^nes. " " * 

In order to construct this transition diagram, all actions 
except ^specific control operations were deleted from the behavior 
stream . > Thus , for example , activities of locat ing ( touching) a 

* conti^ol , ' 'or looking at various parts of the device were dropged ' 
^ froiB the analysis.^ The resulting sequences were then ^subj ected to 

"a sprting process in "which conraion sequences we^^e identifieJT apd , 
then*' the data regrouped ' according -to the 'sequ^?K5^s, and ;fehe 
process rj^peated until no more sequences could be formed^. 

Once these sequences were defined, the behavior patterns for 
■all of the subjects could be rewritten as a tree diagram, in which ' 
all subjects begin at- the origin and then branch out aqcording ^ to ' 
^the first action or sequence subroutine that th'ey perform^ ^and 

* ^hen branch out fur,ther depending on the*ir ind^^vidual Actions. 
-'Since all subjects eventually did sdSie; action that was different' ^ , 

from that done by any other sub ject , ^ eventually the trees all had 

* the &ame number of branches as there were subjects., 

Pattern results . In Figure 2 is shown the top^ level ^ diagram 
for the ^ sequences for the nonexperts knd experts on ttiTe'radio. 
Notice how the nonexpert network seems to be "bushi^er" than the 
expert^ network, and ^Iso appears, to have mor^ d if ferejit 
. subroutines^ Beyond the preference for initially plugging in and . 
turning op^ the- radio^ there seems to be little ifi the way of art 
iTiterpretable pattern ^n the nonexpert sequehces. However, therfe 
is a basic pattern to the exper* sequences. The subjects who./, 
followed the- bottom two major branches first "set ujd" some portion/ 
^ of t^he ^ radio before turning it on. The. subj ect? following th^^ 
, upper branch turned pn the ra4io immediately and then proceeded to y 
make a series of adjustments t<j it. Thus, even with as simple a / 
/ device as a radio, there seem to be two major methods of operating /' 

^ it; the- first is se'feting it up and then turning it oh, followed 
■ . ■ ' 
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Pc'plu^ in power cord 
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Haaretract antenna ' 



Flguje 2.. Behavior sequences for expert and none^fpert isubjects 
^ operating the radio without instructions. ^ 
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*'by adjusting it, and the second is turning it on, then- setting 'it, 
up and adjusting it. Within each of these two major patterns^ 
there are many minor, variations*' / ^ " ^ 

---"'^ A-,similar apparent difference betwee^i experts and nonexperts. 
^ appears with the cassette' recorder in Figure 3*- Overall, the 

experts appear to produce shorter and simpler sequences then' the 

nonexperts. Thus^ the experts in beth the radio ai^d the tape 
. re"coVder ajjpe^ai^, to ' have moi;e consistent and shorter behavior 

sequer^ce^*' Some quantitative compajrisons between . the expert and ^ 
. nonexpert transition net'works were very , intriguing, bu$ none^of 

1>hem reached .statistical significance. ' " 

1% should be noted that, some of the complexity of .the tajie' 
recorder 'behavior sequencee^^is /probably due" to the fact that the 
tape cassette, .was. del iberately given fo the subjects in a 

* condition in which it^was not fully rewound. Since the subjects'" 
task was to record *'testing 'one-tw6-three;' on^the '^^pe and play'^it 
back, this confused some subjects if they rewound the tape all the* 

^ way bapk afte^ rec<irding as one, normally would. , some^ 

subjects, evpn experts^ had ^ to make more^than ©neMttempt to 
recor^T^he ■ tape . Perhaps this complexity is a reflection of the 

^fact that -the HSpe reciorder wis not left in a schemal^ic" stat^; ^ 
that is, the normal'-sta^e foi; a tape cassettg.is that*it .is fully * 
r^ev^ound.vc ' ■ ^ . 

>An import$.nt conclusion is. th^t if ^"Wffere ^ is an 

* difference betw^rf experts and nonexperts, even ^ these 
dev^ices-, theh expei^ta are Tietter even at operating 



d^vi.c^s ' than nonexperts. This prt 
future studies' of electronics expertii 
strongly that nahekpjerts Can- not. 
stu*dies eve/i if very familiar- devices 



•appareji^ 
'everyday 
everyday 

^sents avserious problem -for 
re, because it suggests very 
be used a$ subjects t)f such, 
are used; - * * 

A further ^result that follows from e^rr exstmination of ; thes$ 
two networks, and was also clearly apparent with^^the otiie'r devices 
is t^iat ther^ is' in fact very little stereotypy^ ^ in tfi'e * specific 
behavior sequences. Figure 4 presents th^ transition network for 
the five experts" using the VOM. Notice tha^ the number, of 
subroutines ig quite small, and there is an almost inunedtate 
branqhing of the tree into unique paths, one 'for each subject. * 



Because of tije ' extreme length of th 
oscilloscope and signal generator coi^binatio 
truncated and condensed version 'of the full 
For ' example, the term CRT means any contro 
d^ with adjusting the CRT trace on the Oscil 
involve any sequence of the five contro'ls. 
any sequence involving adjustments to * -the 
base, which alsa involved several control 
that even after this extreme condensation th 
network again branch into unique patterns 
phenomenon demonstrator in Figure 6 also sho 
branching into unique paths. 



e • sequences for the 
n. Figure 5 presents a 
transition^ network. 
1 activities having) to 
Iosco pe, which could 
Likewise, TB refers to 
oscilloscope * s time 
s. It should be noted 
e pa th 3 th r 0 ugh th e 
very quicklyr* The phi 
ws a relatively quick 
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HOflMPERTS 



STAHT 



>[RpTk3t-Rif3tl 

^[prPc-B3IcCl]^PlLe 

V[[(RpTk3t-R¥3T)-(PlL8-V13t)]-RifSt} 




PlLe—Vl — Pf Pits Rv PlLs — ?t — *Rvgt-[?l3t-([RpTk3t-Rw3JtJ-PlL83t)]^-= STOP 

[Rw37^t[(RpIJc3t-RM3t)-PlVlL83tl-Jlif3t]} ■ STOP 

-[Plt8"St] [(RpTk3t-R¥3t)*-(PlLs-V13t)} : ^ STOP 



[R¥St4[(RpTk3t-Rif3t)-PlVlLo3,t]-R¥3t]] 

Ic--Rif3t P13t [RpTk3t-Vl] P13t- 



•RpTk?t Rif [PlLa-V13tJ-^Rif- 



EJIcCl- 



'Rc* 



-Tic — 3t* 



-[?lLo-V13t} 
Ic— CI PC 



[Rv3t-C PlL8-3t)] VI Rest — 

Vl"-[[(RpT1^3t-R¥ST)-;CPlLp-V13t)l-flM3tl 

{P13t-£[RpTk3t-Rv3tJ-nt83t)l-77UrRpTk8t-Rif3t)*(PlL8-n3t)]*Rvgt 



-HL(Hp^*c3t-RM3T)*(PlLB*VlSt)J-R¥3t> 



-3T0P 
STOP 

-S^OP 

-STOP 





[(RpTk8t*-Rv3t)-( PlU-V13t)h 
[RpTk3t,-Vl] RvSt — PlVlL83t-*-[Rif3t-(plL8-3^)] [(RpTk3t-RM3t)-PlVlL83tl STOP 



-RM3t [PlL8*(St*R¥St)J (PlI^-Sth[(RpTk-^(St-Rl,St))-(PlL8r(3t-Rlf3t))Hfl7^PlI*Sr^l3tl-H¥— ^-S^ 

-CI [23-IcCl] i'(Rprk-(3t-Rw3t))-(pll^-(3t-RM3t))H(RpTM31^R¥St))-(plL8*3t)y s4o? . 

[E3*IcCl] ■ [(RpTk-(3t-Bif3t))-FlVlLe3t] ' 3T0P 

Fl— 3t--PrFc-{RpTk*(3t-R¥3t)] [(B3-IcCl)-Rif^t] [(PlVlL«-(8t-RvSt))-(PlL8-VlSt)l— [BJ-IcCl^' 

> ^ — £(ll^Tk-(3t*RM3t))-tPlLa-3t)]^ ' ^TO? 

PrPc [(RpTk-(^t*Rif3t))-(PlL8*3t)) ^ STOP 

[PrPc-(E3IcCl-Rif3t)] [RpTk-(3t-Rif3t) J [( PlVlLfi*(3t-RM3t))-(?lL8-8t)) ^ 3T0P 

l}^ [?rPc-(B3IcCl*R¥3t)] [( PlL8-V13i)-Rif3t]*[(RpTk-(3t*RM3t)H(PXLa-V18t)-Rifat]-^ #— BTOP 

Pc ?r :-[E3IcCl-R¥3t] ^[(RpTk-{3t*Hif3t))*?lVlLe3t) ^3T0F • 




■PrvpIUg cord Into recorder 
RwArewlnd key 
Vl'svoruoe control 
01=open 11^ 



?c»plttg cord into outlet^ -Ic«tn8ert caaaette 
St'atop key Plsplajr key 

ta^Iisten to aound .Clacloaa lid' 

Lbvlook Into batterjr coDpartsent 



Ejvftjftct bttttoa 
Rp»rncord ^ play keya^ 
Tkvtaljc Into recorder 



Figure 3, Behavior sequences for expert, and nonexpert Objects operating 
the cassette tape recorder^ 
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t 

h 



— Ir* 
'lb' 

STARTS 



• Tc 



-Ic — lb — 
CI — TcTa- 

-Tm — CI 

-Tai Irlb- 

-Cl Ui- 



-Cl 
-Tq 



-Ul — CbCrRn 



l,blr CI Up- 



-Iblr^ 



Ir»lnoert red l«ad 
Cr^clip red lead. to reeidtor 
Or^unciip red lead froo resistor 
A;a=adjudt jaeter £ero ocrev 
?z=turn sero ohias knob 



— [ClTa-aiCrCb] 
Ib=inaert Uack lead 




-[tJlAs-ClTg] Tc — Ta — tJl CI- 

TcTa Ao [aiAs-ClTa] Ul— Am — ClsCrRn 

ClTa — CbCrRn 



'lEc— rJhSll^lJlCf Cb 3 — Ux— Ad— 
-rc:7^Ur — ClTsj— Ul — CbCrRn — STOP 



l^clip leada tofiather 



Cb'Clip black lead to resistor / ^Isuncllp leads ^ 
Ub^uncllp 1)lack lead frcti refii9:^o^^xi(imcllp both leVdlp froa resistor 
Tc^turn center rajige selector N tf&sturn taode selectpr 
Rit'raad n\ui1)sr froa neter scale 



Figure 4« B&havior sequences for experts operating the Vol6rOhin**Milliajneter^ 




EXPERTS 



GEIf COH— CRl 

CRT — VI- 



» CRT TB ^ 



CKT — TB CRT TB VI 

^VI TB CRT 

Tfi CRT TB CRT — TB . * . 

'COB*=— VItt- con CRT * . . 

CKT TB CRT . - . 

both COtl GEH — COB — Turn-on-ooc 

Tover-*up-gen OEH— Power -up-oac CKT — COH — CKT TB 



a£N=adjust generator controls CRT^adJiiQt CKT controls TB»adjuQt tloe base oontrols 

Vl=adjust vertical Input .controla COH=oaSe cdnnectloaa 
power All=pover up both unltg Pover-up=plug In olid turn on 



Figure 3. Highly condensed behdvior sequences for experts operating 
the oscilloscope. 
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Lack of fixed procedures. The fact that on the whple there " 
is very Tittle stereotypea- behavior seems to d isconf irm^ the . 
hypothesis sugges'ted above, whi^ch is that device schema knowledge ^ 
tightly specifies operating procedures for devices. However, i'u 
sriojald be pointed out that there are, some stuong consisteno-ies in ^ 
at least the initial s1;ages of operating at least some of the \ - 
devices. For example, with the radio {Figure 2 ), all sub;3ects \ 
plugged i*t in first." With the 'recorder, roughly ha.lf of the \ 
experts ana nonexperts plugged the device in as the first s1;ep. 
With the oscilloscope, most of the subjects plugged in and turned 
on both the oscilloscope and the signal generator before going any ' 
further, but there were some subjects that performed only part^of 
this operation before proceeding. Likewise, not,ice that many 
subjects, after performing- the power-up operation^, went on to 
connect the two devices together before proceeding, any further. 
Finally, with the demonstrator, again mOst of the subjects plugged 
in the cord first, although some of them pj^ug in all of the cords ^ 
and connectors before turning on the device.* 

The VOM presents an interesting contraslJ^ because it does not - 
have to be, plugged in and turned €n . Notice thai there is very 
little stereotypy in the sequence of activities. One might' think 
that inserting the test leads would be the natural first step, but 
only two of the five subjects did this. Or one might think that . 
adjusting the meter to sero would be a natural first step; only 
one of -the subjects did so, although it should be noted that this 
is not a routine operation in the normal use of a meter of this 
type. Thus, it appears that there is some stereotyped behavior, 
but it is limited to some of the very initial stages of device 
operation, and concerns mainly "power-up*' procedures. If people 
indeed follow schematic procedures, these procedures are of such a 
limited and varied nature that characterising them as. schemas is 
of little value . . ' . 



How subjects operate from memory . This lack of stereotypy 
requires explanation . Closer examination of the task situation of 
'operating a device from memory suggests that the expectation that 
device operation would show stereotyped orders^ is not reasonable. 
That is, although the device^ were representatives of a very 
familiar type of device, sixch as a radio, the likelihood, that an 
individual^ subject had' actually had extensive practice with 
operating this particular make and modisl of device is essentially 
sero. To some extent, every device was a novel device to every 
subject* Thus, none of the actual skills of operating the device- 
would be highly automated, becUuse this would only be the case if 
one were familiar 'with the specific location and p^bperties of the 
particular device, y Thus, subjects were essentially operating 
these devic.es in ^ joroblem-solTing mode, instead of a memory 
retrieval mode. Once the problem is looked at in this light, the 
lack of stereotypy "in the behavior becomes clear. 

In ai>y actual device, there sn^e constraints that are imposed 
by the device on the order in which things are done. For example, 
on an oscillosco|)e, the intensity control must be adjusted before 
a .trace oan be seen, and the oscilloscope .can not.be used until 
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53/ 

Pa — ^tPrTd-Pa]- 



\ 



[Td-PrTdl- 



[PrPa( Td-PrPa) ] PrPa- 

ClPa : ClPa 



» ClPa- 
-Cl- 



[Td-PrPa3* 

IdPr- 

PrPa 



-FrPa [PrPi(Td-PrPa)]- 

-PrT<i ; 



-Pap- 



-PrPa- 



tTd-BrTd]- 



Td^turn 



in power cord 
dial 

power off 



Posturn power on 
PaxPlip A/fi QOde 



Pa^plug in l^^t A 
Pr=Plip ra^phase 




Pb>plug %n light fi 
Ql:ich6Ck iighta 



-STOP 
STOP 
-STOP 
-STOP 
-STOP 
-STOP 
-STOP 
-STOP 



Figure 6. Behavior sequences for experts operating the phi phenomenon demonstrator. 
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•the trace is visible. Before intensity control can be 

properly^ adjusted, however, the oscilloscope must be turned on. 
Thus, for any device, there are some "constraints on ^he order of 
certain ^operations. However, even fo*r relatively simple devices, 
such as A radio, th^se constraints in fact specify very little o-f 
the exact order of operation; many steps are independent of order 
given that the overall constraints are met. For example, the 
radio tuning can be adjusted at any tim^, but most us,efully after 
the radio is audibly playing. Thus, referring to Figure 2, there 
are many differerrt orders in which the expert subjects operated 
the controls on the radio, and- there is a uniq^ue path for every* 
subject. "However, all of the subjects succeeded in operating the 
radio, and typically with very little wasted time'or steps. 

Conclusion . The best characterisation of operating a piece 
of - eq^uipment from memory' seems to be tha*^ .subjects perform 
'problem-solving by .deterDjfining what constraints need to be 
satisfied along the way,, and then operating the controls in^a 
manner that meets the constraints and accomplishes the task, but 
does not necessarily follow^ any prescribed ' order . Since a major 
constraint is that the device be operating before it ^ can be ^ 
adjusted, there is a strong tendency for "power-up" stepS to be 
done^first. Since these data involve only a single observation'on 
each 'Subj ect ir^ each device^ it is impossible to tell whether each 
subject was .following an individual stereotyped sequence, which 
seems unlikely. However, it is very clear that device operating 
sequences do not have a *major property of schemas, namely ^ 
stereotypy of content. ^ 

xhe larger implication of this conclusion is that even though 
experts can operate even -complex pieces of equipment completely 
from prior knowledge, they do not perform this by rote memory 
retrieva]^ but rather by a very general problem-solving process. 
For example, the best characterisation of what the experts did 
with the oscilloscfope is that once they had it plugged in, turned 
on and connected with a signal generator ^ *they made many passes 
over the controls making various fine adjustments in ajl' sections 
of the oscilloscope until they had * achieved the final desired 
result. . Many of the operations were undoubtedly . redundant from a 
str icjtly technical point .of View. However , these general 
processes are powerful enough that the experts* could operate the 
completely novel device, the phi phenomenon demonstrator^ without 
any instructions, and quite often without any serious mistakes or 
wasted actions. 

The general conclusion is that expertise does not consist of 
a set of canned procedures for operating different devices'^ but 
rather of a set of' powerful problem-solving heuristics which can' 
be applied to even novel devices^ but which are not very efficient 
even wittf familiar devices. 




Page 18 

SUlilMARY' ^ ^ 

The introduction listed three question^ -that this experiment 
was designed to address. These concerned the instruction format, 
the nature of expertise effects, and the nature of the prior 
knowledge that people would have about electronic eq^uipment. This 
experiment yielded information about each of ti^ese three questions 
which can be summarised as follows: , 

Instruction Format . Contrary to intuition > the menu format 
was not better overg,ll than the ^tep-by-step format; which format 
is ' supef io'r depends on the user ' s experience . Under , some 
conditions the specific experience with the actual de.vice involved 
can be more impoTtaht than the user's general expertise. If the 
device is faailiar, the menu format helps, as would be expePted, 
by r^^iiUcing the amo^unt of instructions ^hat must be read . 
f5ub;Sects tend not to read familiar steps ^ch as descr ijtiojis of 
'how to power-up the equipment which everyon*e knows, nor do they 
read descripti ?ns 'of procedures that are very similar to ones they 
have ^^st-oompleted . tx a device is not familiar, the user can go 
astray, and the' result may be much worse than using step-by-step 
instructions in terms of total completion time. 

. 

' Expertise Effects . Expertise had^both specific and general 
effec^&s in ^^lese results. Experts were faster overaU-, both in 
•the menu and tie step-by-step conditions. But experience with the 
specific device can be as important as the general experience. 
The experts were more efficient than the nonexperts in terms of 
being abtfe to Darry out complicated physical activities. Although 
everybody knows certain things stbout electronic equipment, such as 
how to turn on a device, even on everyday devices the experts ar^e 
more efficient and more consistent in their activities th'an 
nonexperts - * ^ • 

Prior knowledge of electronic devices . It was proposed that 
since people apparently have schema Enowlec^jge---..^^^ 

vices , ^ that they would also have knowledge /of schematic 
procedures for operating devices. A primary chWecteristic of 
such' schematic procedures would be ^ J high degree , of 
stereotypicali fcy in^ how the devices were operated when sub;Sects^ 
did not choose to read instructions. This expectation was 
contradicted oy the ^ data; .there was very little stereotyped 
behavior when subjects operated the devices strictly ^n Jfche - basis 
of tlieir prior knowledge. ^ ^ ' * 

A more accurate assessment is based on making a distinction 
between what people do when they have a highly automated skill at 
operating a particular piece of equipment, and the' ability to 
operate equipment in a more 'normal setting in which every piece of 
equipment is familiar, but not highly practiced. In this case, 
what subjects do is to engage in complicated problem-solving 
strategies , wliere the individual operating steps meet loos^ 
constraints tnat are imposed by the nature of the /device > .but do 
not otherwise fall into a strict stereotyped sequence. This 
problem-solving strategy is very Robust but it is inconsistent 
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between inaiviauals and can be inefficient. Experts clearly have 
much more powerful strategies 'than nonexperts for operating 
devices on the basis only of prior knowledge, but in the case of 
unfamiliar eqijipment , their performance may actually be 
considerably poorer than that of nonexperts who are following 
strict * step-by-step ins true tlon8>. 



References 



Chi, M. 3?. H,, Peltovich, P. J., & &laser, R. {^^Q^). 
Categprizatipn and representation of physics problems by 
experts and novjices. Cognitive Science , 121-^152. 

Chi," M. T. H., & &laser, R., (in press). Problem solving 
abilities. In R. Sternberg' (Ed.), Human abilities:. An 
information-processing approach . San Francisco: Freeman. 

Kieras^ D. E. (1932) . What people know about electronic devices: 
A geggriptiye si;udy (Technical Report No. 12 
TIARZ/DP/TR-82/0NR-12) . University of J^rlsona, Depa^rtment of 
Psychology. 



Pedhasur/ E. J. . '(1932).- Multifile regression in behavioral 



research (2nd ed.j. Hew York: Holt, RinehaXt, S^Winston. ^ 



Smith > E. E". & Goodman > L. (1932). Understand ing, instructions: 
The role of explanatory miater ial (Technical Report No. 50^8^ 
Bolt Beranek and Newman, Inc. , ^ 



/ 



7 



38 



Artzona/Kteras (NR 157-**73) 
Navy 

1 Robert Ahlers 
Code M7n 

Human Factocs Laboratory 
MAVTRAEQUIPCEM " ^ 
Orlando, FL 32813 

'l Dr * Ed Aiken 
' Navy Personnel RiD Center 
San Diego, crf 92153 

- . 1 Df - Meryl S* Baker 

N^y Personnel RiD Center 
San Diego,, CA 92152 

1 Dr* Mick ^nd 
Office of Naval Research 
Liaison Office, F^r E^ac 
APO ^an Francisco, CA Q6503 

1 Dr * Richard Cantone 
Ndvy Research Laboratory 
Code 7510. ^ 
Washington, DC 20375 

1 Dr . Fred Chang 
Navy Personnel RiD Ce/iter 
San Diego, CA 92152 

1 Dr - Stanley Ct^llyeif / 
Office of N^val Technology 
800 N, Quincy Street / 
Arlington, VA 22217 

1 CDR Hike Curran ¥ 
Office of Naval Research 
800 N- Quincy St- 
Code 270 

Arlington, VA 22217 

1 Dr * Jude. Franklin 

Code 7510* 
. Navy RflTsearch Laboratory^ 
' Washington. DC 20375 

1 Dr * Jim Hollan 
Code J 4 

Navy Perscn-iel K i D Ct^nter 
• San Diego, CA 92152 

1 Dr . Ed Hutchins 

Navy Personnel BiD Center 
,San Diego, CA 92452 



25-Jan-8** 



Pa^e. 1 



Navy 



1 Dr* Norman J* Kerr 
Chief of Naval Technical iValnin^ 
Naval Air Station Memphis (75) ^ ^ 
Millington, TN 3805i» ^ 

1 Dr * Peter Kincaid 
Training Analysis i Evaluation Group 
Dept* of the Navy 

Orlando, FL 32813 - . 

1 Dr. William L. Haloy (02) - 
Chief of t/aval Education and Training 
Naval Air Station , » 

Pensacola, FL 32508 

J Dr* Joe McLachlan * 
Navy Personnel RiD C^n^ter 
San Diego, CA 92152 

1 Dr. James McMichael 
Navy PersonnelRiD Center 
San Diego, CA 92152 

1 Dr William, Montague 

* NPRDC Code 13 

San Diego, CA 92152 

1 Technical Director ^ 
Navy Personnel RiD Center 'I'' 
Sgn Di?feo, CA 92152 

fi Commanding Officer 

Naval Research LaboratorjF , 
Code 2627 ' * 

Washington, DC 20390 

1 Office of Naval Research 
Cod e^ 33 ' 
800 tJv-Quincy SStreet 

* Arlington: VA 22217 



6 Persomiel i Training Research Group 
Code WSPT 



1 



Office of Naval Research 
Arlington, VA 22217 

Office of the Chief of Naval Operations 
Research Development i Studies Branch 
OP 115 J 
Washington, DC 20350 



39 



Ari2ona/Kieras(HR 157-473) 25-Jan-84 

. 1 Dr- Robert'G, Sm^fth ^ 
Office of Chiet'of Naval Operations 
0P"987y 

Washirrgton, -DC 20^ * - 

1 Dr, Alfred Smode/oirector 
^ Training A.>iysis i Evaluation Group^ 

Dept. of the Navy^ . , 

Orlando. FL 32813 



Page^ ^ 



t/or 
/T^ Liaise 



Richard Sno^ 
Lson Scientist 
Office of Naval Research 

Branch Office, London 
Box 39 

FPO New York, NY 09510 

1 Dr - Richard Sorensen 
Navy Personnel RiD Center ■ 
San Diego, CA 92152 

1 Dr. Frederick Steinhelser . 
CNO - 0P115 > -> 

Navy Annex 
Arlington, VA 20370 

1 Dr , Thomas Stl^tlt 
NavyS*er3onnel\R4D Center 
^an Diego, CA 92152 

* 

1 Roger Weisslnger-Baylon , 

Department oT'Admln.lstratlve Science? 
.Naval Postgraduate School ' 
Monterey, CA 93910 

1 Mr John H. Wolfe 
Navy Personnel RiD Center 
San DleRO, CA 00^=" 



ERIC 



"Marine Corps 



1 Jin William Greenup 
Education* Advisor (E031) ' 
Education Center, HCDEC 
Quantlco, VA 2213^* ' . 

1 special Assistant for Marine 
Corps Jlatters / 
Code ISDH; , j 
Office of Naval Research 
800 N, Qulncy St/, 
Arlington-, VA- 

1 DR. A-L. SUfTOaSKY 
SCIENTIFIC ADVISOR '{CODE mJ-l ) * 
HQ, .U.S. MARIN^ CORPS 
WASHINGTON. b0 2(fe80 





40 




/ 



Ari2on^Kier*.s (Hh 157-^73) 



25-Jan-8H 



Page 3 



Army 

1 Technical Director 
U, S. Amy Research'Institute for the 
^ Behavioral and Social Sciences 
-5001 Eisenhower Avenue . * 
Alexandria. VA 22333 

1 Dr. Beatrice J. Farr ' ■ ^ m ^ 
U.. S. Ariny Research Institute 
5001 Eisenhower Avenue 
Alexandria. VA 22333 

IDr. Harold F. O'Heil. Jr> 
Director* Training jfesearch Lab 
Array Research Institute < 
5001 Eisenhower Avenue ' * 

Alexandf ia. VA 22333 

1 Commander, U.S, Army Research Institute 
for the Behajfioral i Social Sciences 
ATTN: PERI-BR (Dr. Judith Orasanu) 
5001 Eisenhower Avenue 
Alexandria.^ VA 22333 

1 Joseph 6sotka, Ph,D, 
' ATTN: PEfll-lC 
AViny :iesearch Ins^'ltute 
5001 Eisenho./er Ave, 
Al^exanaria. VA 

1 Dr. Robert Sasmor 
U- S. Arrtl\ Research Institute ^or the 
Behavioral and Social Sciences 
T 5001 Elsenhower Avenue 
^ 'Alexandria, VA 22333 ^ 



1- 



Air Force 



1 U.s/ Air Force 'Of fi'de iif Scientific 
Research 
L^fe. Sciences, Dir.ectorate, ML 

Jlllng Air Force Base 
,ifashington, DC 20332 , 



"T Dr. Earl A. Alluisi 
■•/ 'HQ. MfHRL (ftFSC) 
/ Brooks AFB, TX, 78235 

• 

1 Mr. Eaymond E. Christal 
AFHRL/HOE ' 4 

Brooki AFB, TX 78235 

1 Bryan Da}draan 

AFHRL/LRT 
- Lowry AFB, CO 80230 

1 Dr. Alfred §. Fregly 
, AFOSR/HL 

Bolling-AFB, E»C 20332 

1 Genevieve Kaddad 
Program Manager 
Life Sciences Directorate 
AFOSR 

Boiling {lFB. DC 20332" 

1 Dr. T. M. Longridge 
AFHRL/OTE 

tfilliains AFB. AZ 8522^1 

1 Dr. John Tangney 
AFOSil/HL r 
Boiling AFB. DC 20332 

1 Dr. Joseph^asgjbuke- 
AFHRL/LRT 

Lowry AFB, -CO 80230 



/ 



ERIC 



41 



■Arizoiia/Kieras (IJr 157^^3) 

" l>epa'^t^lent of Defense ^^«^ 



25-Jan-84 



12 Defense Techpic^l* Information Center 
Cameron Station, Bldg 5 - • - v 
.Alexandria, VA 223m- 
Attn: TC , ,^ ^ 

1 Military "Assistant for TrailTing and 
Personnel Technology. 
Office of the Under SeT!T*<Jary of Defens 
for Research 4 Engineering 
Room 3D129f "The pAtagoh 
.Washington. DC 20^01 



1 Hajqr Jack Thorpe . 
DARPA 

UOO Wilson Blv4. * 
Arlington; VA 222C9 - ' 

1 Dr. Robert A, Wisher 
OUSDRE (ELS) 

The. Pentagon, Room 3D129 
Washington, DC 20301 



4<& 



< Page 

Civilian Agencies 

1"f)r. J?^tricia A. Butler 
HIE-BRH Bldg, Stop # 7 
1200 19th St., Ntf 
Washington, DC 20208 

1 Dr. Susan Chipman 
Learning and Development 
National Institute <)f. Education 
1200 19th Street JJW 
Washington, DC 20208 

1 Dr. Arthur Helmed 
72U Brown 

U. S. Dept. of Education 
Washington, DC 50208^ 

1 Dr, An<*rew R. Holnar* ' ^ 

Office of Scientific and Bigineering 

Personnel and Education 
National Science Fowidation 

- Washington, DC 20550 

1 Dr. Everett Palmer 
Hail Stop 239-3 
HASA-Ames Research Center 
Hoffett Field, CA 9^035 



1 Dr . Hary Stoddard . , 
C 10, Hail Stop B296 
Los ^Alamos National Laboratories 
Los Alamos. NH 

1 Chief, Psychological^Reserch Branch 
U. S- Coast Guard (G-P-1/2/TPU2) 
Washington, DC 20593 

1 Dr. Frank Withrow , ■ 

U. S. Office of Education 
UOO Haryland Ave. SW ^ " 
Washington, DC 20202 

^ 1 Dr. Joseph L. Young, Director 
Hemory & Cognitive Processes 
Hatiotaral Science Foundation 
Washington, DC ?O550 



" Arizona/Kieras (MR 157-1(73) 25-Jan-8i| 
I Private Sector 



fage^ 5 



Private Sector , 



1 Dr. John R.. Anderson - 
Department of Psychology 
Carnegie-Hfelloh University* 
Pittsburgh, PA 15213 ' ' . - * 

1 Dr. Alan Baddeley ^ ... 

Hedical Research Council * 
Applied Psychology Unit 
15 Chaucer Road 
Cambridge CB2 2EF 
ENGLAND 

1 Eva Baker 
Director* 

' UCLA Center .for the Study of Evaluation 
1^15 Moore Hall ' 

Uniy^^r^ity of California, Los Anaele's 
Los Angjeles, CA 90O2i| 

Mr. Avrorr Barr 

Department of Computer Science 
Stanford University 
Stanford, CA. 9^305 

1 Dr. Menucha Birenbaum 
School of Education 
Tel Aviv University 
Tel Aviv/Ramat Aviv 69978 
Israel , ^ ' 

i 

1 Dr. John Black 
Yale University * ^ 

Box 11A. Yalp Station 
New* Haven, CI 06526 

1 Dr. John S. Brown 
XEROX Palo Alto Research Center 
3333 Coyote Road ' ^ 

Palo Alto, CA 9113011 

1 Dr. Glenn Bryan ' 
6208 Poe Road 
3ethesda% HD 20817 

1 Dr. Bruce BucltSnan 
Department of Computer Science 
Stanford' Univ^riity 
Stanford, CA 9**305 

1 Dr* Jaifie Carbonell ^ 
Carnegie-Hellofi University 
Department of Psychology* \ 
Pittsburgh, PA T5213 



'^^.l Dr.^Pat Carpenter 

J" pepartment of Psychology ■ 

C ' ? Carnegie-Hellon University * 

. Pittsburgh, PA 15213 

^ 1 Dr. Micheline Chi 
/ ' Learning R i D Center . ^ - 

♦University of Pittsburgh 
3939'0*Hara- Street ' ^ 
Pitt;sburgh^ PA 15213 

1 Dr. William Clancey . 
m ^Department of Computer Science 
Stanford University 

Stanford, CA ^1I306 

\^ - 

> 1 Dr. Michael Cola - 

University of California 

at Saff Diego 
Laboratory (5f Comparative 

Human Cognition - DOO3A 
La Jolla, CA 92D93 

1 Dr. Allan M. Collihs 
A Bolt Beranejki Newman, ^Inc. 
50 Moui ton -Street ^ 
Cambridge, MA 0213^ 

1 Dr. Kenneth B. Cross, 
Anacapa Sciences, lnc« 
P.O. Drawer Q , 
Santa Barbara; CA 93102 

1 Dr. Emmanuel Donchin * 
pepartment of Psychology 
- ' University of Illinois 
Champaign, IL 6I82O 

1 Dr. Thomas H. Duffy 
Department of English* 
Cjarnegie-Hellon Universi^ty 
^henley Park 
Pittsburgh, CA 1|;?13 

1 ERIC Facility-Acquisitions 
^833 Rugby Avenue 
Bethesda, MD 2601i| * 

1 Dr, Anders Ericsson 
* Department of Psychology 
University of Colorado 
^ Boulcfert CO 80309 



ERIC 



43 



Private Sector ' » 

1 Dr. Paul Feltovlch 
' Department of Medical Education 
Southern' Illinois University 
' , Schoorof Hedicine ' ^ 

, ^,0. Box 3926 ^ 
Springfield, IL 62708 , 

1 Professor Heuven Feuerstein 
(WCHI Rehov Kannon 6 
Bet Maker em ^ 
Jerusalan , 
Israel 

1 Hr. Wallace Feurzeig 

Departpient of Educational Teclinalogy 
*Bolt Beranek 4'Hewraan 
10 Houlton St. ^ 
' - Cambridge; HA 02238 

1 Dr. I^ter Fletcher 
University of Oregon 
Department of Computer Science 
.Eugene, OH 97^03 

^ t *Dr<'John H* Frederil^j^**" 
Bolt Beranek. & Neman 

5£f Houlton Street , ^ 

7 Cambriage, HA 02138 
. - 

^\ Dr. Hichael Genesereth 

Department of Computer Science 
Stanford University . 
Stanford, CA 9^305 



Page 6 



1 Dr. Don Gentner 
^enter^^for Human Information Processing 
University of* California, San Diego 
la Jolia, CA 92093 



.1 Dr. Dedre Centner 
Bolt'Beranfek & Newman 
10 Houlton St- 
Cambridge, KA 02138 

1 Dr . Robert^ Glaser 

learning Re^search & Development Center 
Ui^iversity of Pittsburgh 
3939 O'Hara Street 
PITTSBURGH, t'A 15£60 

1 Dr. Harvin D. Glojik 
217 Stone Hall. 

Cornell' University - , ^ 

Ithaca t NY 14853 



pHvate Sector 



1 



1 Dt^. Josph Goguen 
SF^ I^nternational 
^ 33& RavensMOod Avenue// 
,^ He^lo Park, CA 9t|02^ 

1 Dr], Daniel Gophen 
Faculty of Indusirial Engineeri 
& Management 
< TECHNION 

Haifa 32000 - 
ISRAEL 



1 Dr. fiert Green ^ 
Johns Hopkins University 
Department of Psychology 
Charles & 3Hh Street 
Baltimore, HD 21218 ^ 

1 DR. JAMES G. GREENO 

LRDC . ^ 

UNIVERSITY OF PITTSBURGH 
3939 O'HARA STREET * 
PITTSBURGH, PA 15213 

1 Dr. Barbara Hayes-Roth 
Department of Computer Science 
Stanford University - 
Stanford, CA 95305 

^ Dr. Joan I* Heller 

Graduate Group in Science and 
* Mathematics Education 

c/o School of Education 

University of ealiforni^^ 

Berkeley, CA 9il720 

1 Dr. James R, Hoffman 
Department of Psychology 
Urriversity of Del-?ware 
Newark, DE l97l 1 " , 

^ * 

1 American Institutes for Research 
1055 Thomas Jefferson St,, N.W.* 
Washington, DC 20007 

1 Glenda Oeenwald, Ed, 
Human Intelligence Newsletter 
P, 0, Box 1163 
Birmingham, HI i|8012 

1 Dr. Earl Hunt 
Dept. of Psychology 
University of Washington 
Seattle, WA 98105 * 



ERIC 



M 



Arizona/Kieras (HR 157-473) - ' 

Private Sector 

1 Robin Jeffries 
* Computer Research Cenfifer 
'Hewlett-Packard Laboratories 
' 1501 Page Mill Road . 
Palo Alto» CA 9t30H 



25-Jan-8i( 



Page 7 



1 Dr . Marcel Just 
Department of Psychology 
Carnegie-Mellon University 
PittaburghT PA 15213 



1 Dr. Wal^ter^Kint^Scri 

Department of ^sycho^ogy 
■ University of Coloraoo 




1 Dr. DJTt^t<Klahr ^ 
Department of Psychology 
Carnegie-Mellon University 
Schenley Park 
Pittsburgh, PA 15213 

1 Dr , Stephen Kosslyn 
123^ William James Kail 
33 Kirkland St. ' ^ 
Cambridge, MA 02138 

1 Dr , Pat Langley 
The Robotics Institute 
Carnegie-Mellon University 

- Pittsburgh, PA 15213 

1 Dr . Jill Larkin 
Department of Psyctiology 
Carnegie Mellon University 
Pittsburgh. PA 15213* 

l^r, Alan Lesgold 
Learning RiD Center 
University of P-L*.i.:*ourgh 
3939 O^Hira Street 
Pittsburgh, PA i5260 

1 Dr . Jim Levin 
University of California 

at San Diego 
Laboratory fof Comparative 

Human Cognition - D003A 
La Jolla, CA 92093 



Private Sector 



\ Dr. Michael Levine: 
Department of Educational Psychology 
210 Education Bldg. 
University of Illinois 
Champaigrr, IL^l801 

1 Dr. Marcia C. Linn 
Lawrence Hall of Science 
University o€. California 
Berkeley, CA 9^720 



1 Dr. Don Lyon 
AFHRL/OT (UDRI) 
Williams AFB, AZ 8522' 



1 Dr. J3y McClelland 
Department of Psychology 
MIT , . 
Qambridge, MA 0213^ 

1. Dr. James R. Miller 
COTiputer»Thought, Corporation 
1721 West Piano' Highway 
Piano, TX 75075 

1 Dr. Mark Miller ^ 
Computer»Thought Corporation 
1721 West Piano Parkway 
Piano, TX 75075 

1 Pr. Tom Moran' 
Xerox PARC 

3333 Coyote Hill Road 
Palo Alto, CA 9t30H 

1 Dr. Allen Munro 
Behavioral Technology Laboratories 
181(5 Elena Ave, Fourth Floor 
Redondo 'Beach, CA 90277 

1 Dr. Donald A Horntan 
Cognitive Science, C-015 
Univ. o*' California. San Diego 
La Jolla, CA 92093 

1 Dr. Jesse Orlansky 
Institute for Defense Analyses 
1801 H. Beauregard St. 
Alexandria, VA 22311 



Ik 



45 



ERIC 



i 



Arlzoua/Kleras (hr 157-U73) 

* 

Private Sector 



25-Jan-84 



1 Dr. Hincy Pennington 
University of Chicago 
Graduate Schoo? of Business 
1101 E. 58th St. 

'Chicago,, IL 60637 — , 

%' DR. PETER POLSOH 
DEPT. OF PSYCHOLOGY 
^ UHIVERSITY OF COLORADO 
BOULDER, CO 80309 

1 Dr. Hike Posner 
Department of Psychology 
University of Oregon 
Eugene, OR 97^03 

1 Dr . Lynn Reder 

Department of Psychology 
CarneglerHellon ,Un Iverslty 
Scheniey Park 
Pittsburgh, PA 15213 

1 Dr/ Fred Relf £0 
Physics Department 
University of California 
Berkeley, CA 9^720 

1 Dr. Lauren Resnlck 
LRDC 

^ University of Pittsburgh 
^ 3939 0»Hara Street 
Pittsburgh, PA 1521 

1 Dr. Jeff Richardson 
Denver Research Institute 
University of Denver 
Denver, CO 80208 

V Mary S- RHey 

Program In Cognitive Science 
"^nter for Human Information Processing 
University of California, San Diego 
LaJoIIa. CA 92093 

1 3r . Andrew Rose 
American Institutes £or Research 
1055 Thomas Jefferson St, NW 
Washington, DC 20 007 

1 Dr, Ernst Z. Rothkopf . 
Bell Laboratories 
Hurray Hill, Hj 07971 



Page 8' 

Private Sector 

1 Dr. William B. Rouse 
Georgia Institute of Technology 
School of Industrial i Systems 
' Engineering' 
Atlanta, GA 30332 

1 Dr. David Rumelhart 
Center for Human Information Processing 
-Univ. of California. San Diego ^ 
U Jolla. CA 92093 ^ 

1 Dr. fflchael J. Samet 
Perceptronlcs, Inc ' 
6271 Varlel Avenue 
Woodland Hills. JUL 91 36^ 



1 Dr- Roger Schank 
Yale University 

Department of Computer Science 
P,0. Box 2158 
•Hew Haven, CT 06520^ 

1 Dr. Walter Schneider 
Psychology Department 
603 E. Daniel 
Champaign, IL 61820 

1 Dr, Alan Schoenfeld 
Mathematics and Education 
The University of Rochester 
Rochester, HY 1it627 

1 Mr. Colin Sheppard 
Applied Psychology Unit 
Admiralty Marine Technology Est. 
Teddlngton, Middlesex 
United Kingdom 



1 Dr. H, Wallace Slnalko 
Program Director 

Manpower Research and Advisory Services 
Smithsonian Institution 
801 Horth Pitt Street 
Alexandria, VA 22314 

1 Dr . Edward E* Smith 
Bolt Beranek & Hewman^ Irfc* 
50 Houlton Street 
Cambridge, MA 02138 



ERLC 



46 



zona/Kieras XHR 157-«73) " 25-Jan^4 



Page 9^ 



Private Sector 



Private Sector- ' 



1 Dr. Eliott Sol6way 
Tfalej- University 

IJepqrtment of Computer Science 

P,0. Box 2158 

New Haven, CT 06520 

1 Dr, ICathryij -T. Spo&hr 
Psychology Hnpartnent' 
Brown University 
Providence, RI 02912 

1 Dr, Robert Sternberg 
Dept. of Psychology 
Yale University 
Box IIAr Yale Station * 
New Haven* CT 06520 

1 Dr^ Albert Stevens 
Bolt BeraneK & Newman, Inc, 
10 Houltop St. / 
Cambridge, MA 02238 

1 David E. Stone^ Ph.D, 
.Hazeltine Corporation 
7680 Old Springhouse Road 
McLean ♦ VA 22102 

1 Dr, Kikumi Tatsuoka 
Computer Based Education Research Lab 
252 Engineering Research Laboratory 
Urbana. IL 618OI 

1 Dr. Maurice Tatsuoka 
' 220 Education Bldg 
1310 S, smh St. 
Champaign 61820 

1 Dr', Perry W, Thorndyke 
Perceptronics, Inc, 
5^5 Middlefield Road. Suite 140 
Menlo Park. CA 94025 

' Douglas TPwne 
Univ. of So, California 
Behavioral Technology Labs^ 
1845 S. Elena Ave, 
Redondo B^aoh* CA 902'^'^ 



1 Dr. Keith T. WeScoUrt 
Perc eptronlcs # Inc . 
545 .Hlddlefleld Road, Suite 1ilO 
Henlo Park* CA 94025 

1 William B. Whitten 
Bell Laboratories 
2D-610 

Holmdel, HJ 07733 

1 Dr. TJjjMoas Wickens 
Department. of Psychology 
Franz Hall 

\ University of Cal^ifornia 
405 Kllgarde Avenue 
aos AngeleSt CA 90024 

1 Dr, Mike Williams 
Int^llGenetics 
124 University Avenue 
Palo Alto^ CA 94301 

1 Dr, Joseph Wohl 
Alphatecht Inc* 
2 Burlington Executive Center 
111 Middlesex Turnpike 
Burlington* HA 01803 



Dr. |Kurt Van Lehn 
Xerox PARC 

3333 Coyote Hill Road 
Palo Alto. CA 94-304 



47 



